Ergodic multigram HMM integrating word segmentation and class tagging for Chinese language modeling

A novel Ergodic Multigram Hidden Markov Model (HMM) is introduced which models sentence production as a doubly stochastic process, in which word classes are first produced according to a first order Markov model, and then single or multi-character words are generated independently based on the word classes, without word boundary marked on the sentence. This model can be applied to languages without word boundary markers such as Chinese. With a lexicon containing syntactic classes for each word, its applications include language modeling for recognizers, and integrated word segmentation and class tagging. Pre-segmented and tagged corpus are not needed for training, and both segmentation and tagging are trained in one single model. In this paper, relevant algorithms for this model are presented, and experimental results on a Chinese news corpus are reported. Another approach is to keep all possible segmentations in a lattice form, score the lattice with a language model, and finally retrieve the best candidate by dynamic programming or some searching algorithms. N-gram models are usually used for scoring [6] [7], but their training requires the sentences of the corpus to be segmented (and tagged if class-based N-gram is used [7]). We introduce the Ergodic Multigram Hidden Markov Model which, when applied as a language model for these languages, integrates the segmentation and tagging processes into one model, and does not assume any prior segmentation or class tagging. Thus both training and scoring can be done using the model directly on a raw corpus. This model can be applied as a language model for an input sentence (or a character lattice), and the maximum likelihood segmentation and class-tagging of it can be obtained using the Viterbi or Stack Decoding Algorithm. 2. TERMINOLOGY